Skip to content

Refactor to continuous background preimage generation and L1 finality caching #30

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yoshidan merged 50 commits into from
Jan 13, 2026
Merged

Refactor to continuous background preimage generation and L1 finality caching #30

yoshidan merged 50 commits into from
Jan 13, 2026

Conversation

@yoshidan
Copy link
Collaborator

@yoshidan yoshidan commented Dec 6, 2025 •
edited
Loading

Description

This PR significantly refactors the architecture to switch from on-demand preimage generation to a continuous, always-on background generation model. This improves response latency and ensures preimages are readily available for finalized blocks.

Key Changes

  • Continuous Preimage Collection: Introduced PreimageCollector, which runs in the background to continuously fetch L1 finality updates and generate preimages for the range between the agreed L2 block and the finalized L2 block.

  • Decoupled Serving: The HTTP server now serves already generated preimages from the repository (FilePreimageRepository), decoupling the generation logic from the request path.
    L1 Finality Caching: Implementing FinalizedL1Repository to persistently store the latest finalized L1 headers.

  • API Updates:

    • get_preimage: Updated to return pre-generated data from storage.
    • get_finalized_l1: New endpoint to retrieve finalized L1 headers by hash.
    • get_latest_metadata & list_metadata: New endpoints for better visibility into the available preimages.

  • Concurrency & Maintenance:

    • Preimage collection supports parallel execution (max_concurrency).
    • Added PreimagePurger to automatically clean up expired preimages and headers based on TTL.

Naohiro Yoshida added 23 commits November 15, 2025 14:47
bump up kona
5790a5c 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
bump optimism
1b77763 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
bump optimism-package
6bcd7ca
change directory
e6705df 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
change directory
b9eec29 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
start server
32e1baa 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
create preimage sequentially
315ef13 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
configure
6260bbd 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
fix write and load
53356d5 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
collect parallel
fa7566d 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
add test
3797455 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
add test
edc914d 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
fix clippy
1422c66 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
add test
826c641 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
fix conflict
15aebb6 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
add l1 finality cache
7b9ea58
fix
5561b45
fix logic
83c8750
fix logic
816b680 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
fix log
c1db3dc
add test
aa82b66
add inspector
44664ae
add inspection
aaf86cb
@yoshidan yoshidan marked this pull request as draft December 6, 2025 07:12
Naohiro Yoshida added 2 commits December 13, 2025 10:53
tweak
806fb88 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
fix lint
080c36d 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
@yoshidan yoshidan changed the title Enable background preimage making Refactor to continuous background preimage generation and L1 finality caching Dec 14, 2025
@yoshidan yoshidan marked this pull request as ready for review December 14, 2025 01:39
@yoshidan yoshidan requested a review from Copilot December 14, 2025 01:39
Naohiro Yoshida added 5 commits December 14, 2025 12:55
fix format
9809898 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
add unit test for web
c9d920f 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
simplify logic
f027f5a 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
add test
5b9d098 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
use hashset instead of hashmap
84fa27e 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
@yoshidan yoshidan requested a review from siburu December 15, 2025 04:42
change progress on error
fe63460 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
@yoshidan yoshidan requested review from siburu and removed request for siburu December 16, 2025 05:00
fix conflict
9948bdd 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
@siburu
Copy link
Collaborator

siburu commented Dec 25, 2025

@yoshidan I still see the warning: This branch has conflicts that must be resolved.
It is necessary to fix some conflicts, right?

Naohiro Yoshida added 2 commits December 26, 2025 00:10
fix conflict
61b9913 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
tweak
d415e4d 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
@yoshidan
Copy link
Collaborator Author

yoshidan commented Dec 26, 2025

@siburu I fixed the conflict now.

@siburu siburu requested a review from ikehara December 26, 2025 06:10
ikehara
ikehara requested changes Jan 5, 2026
View reviewed changes
Copy link

@ikehara ikehara left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some findings were reported by AI. I reviewed them and commented on the actual issues.

Naohiro Yoshida added 4 commits January 8, 2026 17:16
use modified instead of created
bd07d09 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
use write rename pattern for file
27a49f5 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
reuse http client / add timeout
2541074 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
fix clippy
7816a37 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
ikehara
ikehara reviewed Jan 9, 2026
View reviewed changes
fix ci
2396523 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
@yoshidan yoshidan force-pushed the feature/v2 branch 2 times, most recently from c8d4331 to 5880fe4 Compare January 12, 2026 01:35
rename retry_count -> attempts_count / add the comment
4e2afcd 
Signed-off-by: Naohiro Yoshida <naohiro.yoshida@datachain.jp>
ikehara
ikehara reviewed Jan 13, 2026
View reviewed changes
Copy link

@ikehara ikehara left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any metrics we can check to see if it’s working properly?
I’m concerned that, without metrics, we won’t be able to monitor the progress of preimage generation.

@yoshidan
Copy link
Collaborator Author

yoshidan commented Jan 13, 2026

Are there any metrics we can check to see if it’s working properly?
I’m concerned that, without metrics, we won’t be able to monitor the progress of preimage generation.

No dedicated metrics exist, but we can get the latest preimage metadata through the "/get_latest_metadata" API.
We can perform synthetic checks to see if the value stays the same for too long.

@yoshidan yoshidan merged commit 980eeff into main Jan 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@ikehara ikehara ikehara approved these changes

@siburu siburu Awaiting requested review from siburu

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@yoshidan @siburu @ikehara